With Heptapod 18.6, jump in the managed sidecar

(R)HGitaly and the sidecar

HGitaly is the internal Heptapod service that takes care of all Mercurial repository handling, pretty much like Gitaly does with Git repositories. Both operate by implementing a gRPC protocol of the same name.

As explained in the Heptapod 18.5 retrospective article, we actually have two implementations of HGitaly: RHGitaly is a fast and highly scalable server written in pure Rust, but it cannot perform itself all the tasks specified by the protocol. For the remaining ones, it turns itself to the classical (Python) HGitaly. Being implemented in Python, the latter has access to all high level Mercurial APIs.

RHGitaly today handles a vast majority of the Mercurial work in Heptapod. It is therefore to be thought as the main engine and body. The (Python) HGitaly complements it, whence the "sidecar" terminology to address it.

As a side note and to give due credit, we borrowed this nice terminology from Gitaly, that had for a long time a similar relationship between its core, written in the Go programming language and its legacy implementation in Ruby, originally split out from Gitlab's web application itself.

As of Heptapod 18.5, we had thus two different system services, with all clients calling RHGitaly, while RHGitaly itself was the only client of (Python) HGitaly. Both services need file system access to the repositories, hence have to be in the same system.

This was obviously not satisfying, and actually an obstacle on our road towards cloud-native Heptapod.

Hence we went to the next step: actually removing the HGitaly service by making RHGitaly itself spawn and manage the Python processes. This is what if means for the sidecar to be "managed".

Tuning the sidecar

The managed sidecar provides a much better load balancing and resources management that what HGitaly was previously able to do, but it must be configured according to the available resources.

For Omnibus/Docker instances, this can be done in /etc/gitlab/gitlab.rb or equivalently with the GITLAB_OMNIBUS_CONFIG environment variable. Here is a simple example:

rhgitaly['sidecar'] = {
  min_idle_workers: 2,
  max_idle_workers: 4,
  max_workers: 5,
  max_rss_mib: 1024,
  worker_startup: {
      readiness_timeout_ms: 5000
  }
}

If you change the settings in a running container, the changes can be applied without a full restart with gitlab-ctl reconfigure.

The example above means that RHGitaly will try to always keep at least two Python worker processes ready for incoming requests (hence idle), will start do decommission some if there are more than 4 idle ones, and will never spawn more than 5 workers. Also, if the total resident memory goes over 1GiB, it will mark enough of the largest ones for termination once they are done with their current requests. This is a very soft limit, as it will never interrupt a running request. We may introduce a hard limit in the future.

This rhgitaly['sidecar'] entry maps directly to the [sidecar] section of the RHGitaly configuration file, which is typically located at /var/opt/gitlab/rhgitaly/rhgitaly.toml. More detailed information is given as comments in this file. You can also simply check the RHGitaly configuration file example in HGitaly sources (switch over to the proper version tag if needed).

In a busy Heptapod instance with lots of active Merge Requests, Pipelines etc, the HGitaly Python workers will be requested a lot, and it will happen frequently in bursts. Also, some requests are expected to be fast, while some other have very long timeouts – up to six hours in the GitLab standard. This is why it is important to have idle workers: otherwise the system may have to stack some fast request onto a worker that is busy with a long one.

As you can see with the example readiness_timeout_ms value, starting a worker can be very long, so we recommend not being too shy with min_idle_workers. We also recommend not to set max_workers too much above max_idle_workers, so that it doesn't stop and start workers too frequently.

The sidecar outputs periodic information about the processes in its log. Here's an example on a large instance:

2025-12-15T01:11:28.577883Z  INFO rhgitaly::sidecar: Housekeeping thread: total RSS 2765MiB (configured max 4096MiB)
dumping worker stats before tidying up: [
  WorkerStats { id: 4, pid: Some(3109936),
    status: Busy { count: 1, deadline_in: "56.859178984s" }, rss_mib: 664 },
  WorkerStats { id: 5, pid: Some(510), status: Idle, rss_mib: 777 },
  WorkerStats { id: 6, pid: Some(507), status: Idle, rss_mib: 592 },
  WorkerStats { id: 7, pid: Some(1378), status: Idle, rss_mib: 342 },
  WorkerStats { id: 8, pid: Some(1381), status: Idle, rss_mib: 160 },
  WorkerStats { id: 9, pid: Some(1380), status: Idle, rss_mib: 58 },
  WorkerStats { id: 10, pid: Some(83511), status: Idle, rss_mib: 58 },
  WorkerStats { id: 11, pid: Some(262738), status: Idle, rss_mib: 57 },
  WorkerStats { id: 12, pid: Some(262748), status: Idle, rss_mib: 57 }
]

Right now, filtering the logs for occurrences of RSS is good enough to see only those lines. By looking for rhgitaly::sidecar, you can also witness the spawning and termination of workers.

Versions and delivery history

The managed sidecar was developed with Heptapod 18.6 in mind, both because it was a step in the right direction, and also because we thought it would help with the performance problems of the large instances such as foss.heptapod.net, which we long thought to be due to worker starvation problems.

However, as soon as we started to make perfomance assessments of the managed sidecar under stress, we discoverd that the load balancing of the classical HGitaly service was completely broken. So we were right about the worker starvation, and much more that we thought.

The managed sidecar entered active service early on foss.heptapod.net and heptapod.host, and was backported to the HGitaly 18.5 series. However, it was not in use in the Docker instances yet.

The actual novelty in Heptapod 18.6 is thus that the managed sidecar is now the only mode of operation of HGitaly in Omnibus/Docker instances. Normally, we would make this an opt-in, but it was impractical in that case, because we actually removed the former hgitaly service, and frankly, the managed sidecar is so much better – provided it is correctly configured.